Batería de evaluación de cognición social en psicología forense: adaptación y validación / Bateria de avaliação para cognição social em psicologia forense: adaptação e validação / Assessment battery for social cognition in forensic psychology: adaptation and validation

El objetivo del estudio fue adaptar y validar una batería de instrumentos para evaluar subdominios de la cognición social con población forense en Colombia y México a partir de una selección de instrumentos utilizados en neuropsicología clínica. Se revisó la pertinencia de los subdominios que componen la cognición social como evidencia de validez de contenido mediante la evaluación de diferentes expertos en psicología forense, neuropsicología y medición y evaluación; utilizando el método ANGOFF modificado y un análisis de confiabilidad según el modelo Rasch y el coeficiente Alfa de Cronbach. Posteriormente fue sometido a pilotaje con 20 personas condenadas en cárceles colombianas y mexicanas con privación de libertad. Se encontraron calibraciones INFIT 1.17 y 1.04, OUTFIT 1.08 y 1.18, con índices de separación de 2.66 y 1.63, y fiabilidad de .88 en colombianos y .73 en mexicanos; estos datos proporcionaron evidencias de validez de la batería adaptada. (AU)

O objetivo do estudo foi adaptar e validar uma bateria para avaliar subdomínios da cognição social com população forense na Colômbia e no México a partir de uma seleção de instrumentos usados ​​em neuropsicologia clínica. Revisou-se a relevância dos subdomínios que compõem a cognição social como evidência de validade de conteúdo por meio da avaliação de diferentes psicólogos especialistas em psicologia forense, neuropsicologia e medição e avaliação, usando o método ANGOFF modificado e uma análise de confiabilidade de acordo com o modelo de Rasch e coeficiente alfa de Cronbach. Posteriormente, ele foi submetido a pilotar com 20 pessoas privadas de liberdade sentenciadas em prisões colombianas e mexicanas. Calibrações INFIT 1,17 e 1,04, OUTFIT 1,08 e 1,18 foram encontradas, com taxas de separação de 2,66 e 1,63, uma confiabilidade de 0,88 em colombianos e 0,73 em mexicanos; esses dados forneceram evidências da validade da bateria adaptada. (AU)

The aim of the study was to adapt and validate a battery for the evaluation of subdomains of social cognition with the forensic population in Colombia and Mexico from a selection of instruments used in clinical neuropsychology. We reviewed the relevance of the subdomains that compose social cognition as evidence of content validity through the assessment of different psychologists, experts in forensic psychology, neuropsychology and measurement and evaluation, using the modified ANGOFF method and a reliability analysis according to the Rasch model and Cronbach's Alpha coefficient. Subsequently, a pilot study was performed with 20 people deprived of liberty in Colombian and Mexican prisons. INFIT calibrations of 1.17 and 1.04, and OUTFIT calibrations of 1.08 and 1.18, with separation rates of 2.66 and 1.63, and reliability of .88 and .73 were found in Colombians and Mexicans respectively; these data provide evidence of the validity of the adapted battery. (AU)

Relationships between Bloom's taxonomy, judges' estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study

ABSTRACT BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee / 보건의료교육평가

PURPOSE: This study aimed to compare the possible standard-setting methods for the Korean Radiological Technologist Licensing Examination, which has a fixed cut score, and to suggest the most appropriate method. METHODS: Six radiological technology professors set standards for 250 items on the Korean Radiological Technologist Licensing Examination administered in December 2016 using the Angoff, Ebel, bookmark, and Hofstee methods. RESULTS: With a maximum percentile score of 100, the cut score for the examination was 71.27 using the Angoff method, 62.2 using the Ebel method, 64.49 using the bookmark method, and 62 using the Hofstee method. Based on the Hofstee method, an acceptable cut score for the examination would be between 52.83 and 70, but the cut score was 71.27 using the Angoff method. CONCLUSION: The above results suggest that the best standard-setting method to determine the cut score would be a panel discussion with the modified Angoff or Ebel method, with verification of the rated results by the Hofstee method. Since no standard-setting method has yet been adopted for the Korean Radiological Technologist Licensing Examination, this study will be able to provide practical guidance for introducing a standard-setting process.

Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination / 한국의학교육

PURPOSE: The purpose of this study was to apply alternative standard setting methods for the Korean Medical Licensing Examination (KMLE), a criterion-referenced written examination, and to compare them to the conventional cut score used on the KMLE. METHODS: The process and results of criterion-referenced standard settings (i.e., the modified-Angoff and bookmark methods) were evaluated. The ratio of passing and failing examinees determined using these alternative standard setting methods was compared to the results of the conventional criteria. Additionally, the external, internal and procedural evaluation of these methods were reviewed. RESULTS: The modified-Angoff method yielded the highest cut score, followed sequentially by the conventional method and the bookmark method. The classification agreement between the modified-Angoff and bookmark methods was 0.720 measured by Cohen's κ coefficient. The intra-panelist classification consistency of modified-Angoff method was higher than bookmark method. However, the inter-panelist classification consistency was vice versa. The standard setting panelists' survey results showed that the procedures of both methods were satisfactory, but panelists had more confidence in the results of the modified-Angoff method. CONCLUSION: The modified-Angoff method showed results that were more similar to those of the conventional method. Both new methods showed very high concordance with the conventional method, as well as with each other. The modified-Angoff method was considered feasible for adoption on the KMLE. The standard setting panelists responded positively to the modified-Angoff method in terms of its practical applicability, despite certain advantages of the bookmark method.

Implementing standard setting into the Conjoint MAFP/FRACGP Part 1 examination – Process and issues

The College of General Practitioners of Malaysia and the Royal Australian College of General Practitioners held the first Conjoint Member of the College of General Practitioners (MCGP)/Fellow of Royal Australian College of General Practitioners (FRACGP) examination in 1982, later renamed the Conjoint MAFP/FRACGP examinations. The examination assesses competency for safe independent general practice and as family medicine specialists in Malaysia. Therefore, a defensible standard set pass mark is imperative to separate the competent from the incompetent.

A research on the methods of establishing both domestic and foreign standards of physician qualification examination / 中华医学教育探索杂志

Since the examination of doctor's qualifications came into effect, methods that are set by standard to judge whether one passes the examination or not were in dispute all the while. This article is going to introduce four kinds of methods (Angoff method、Modified Angoff method、Nedelsky method、Borderline group method) which are commonly used at home and abroad. Summarizing their advantages, disadvantages and the usage in the process of standards setting in various countries, we aim to provide reference for setting cutoff score of the examination of doctor's qualifications in China.

The application of Angoff method in setting a standard cutoff score for comprehensive Written Test of China National Medical Licensing Examination / 中华医学教育探索杂志

This paper gives a brief introduction to China National Medical Licen sing Examination(NMLE)and focuses on the definition,modification and application protocol of Angoff Method in setting a standard for NMLE Comprehensive Written Test.Statistic analysis indicates the validity of modifled Angoff Method.Suggestion is provided for future improvement.

Reconsidering the Cut Score of Korean National Medical Licensing Examination / 보건의료교육평가

After briefly reviewing theories of standard setting we analyzed the problems of the current cut scores. Then, we reported the results of need assessment on the standard setting among medical educators and psychometricians. Analyses of the standard setting methods of developed countries were reported as well. Based on these findings, we suggested the Bookmark and the modified Angoff methods as alternative methods for setting standard. Possible problems and challenges were discussed when these methods were applied to the National Medical Licensing Examination.

Setting standards for performance tests:a pilot study of three-level Angoff method / 中华医学教育探索杂志

Dr. Rachel Yudkowsky et. al. have published the paper in Academic Medicine [2008,83 (10 Suppl):S13–S16] demonstrating a reformed Angoff method in which items were assigned to "Yes/No/Maybe" three-level categories for judges to estimate whether a borderline students would or would not accomplish each item,compared with "Yes/No" two-level categories. Five judges were provided three-level Angoff ratings for seven checklists used in a clini-cal skills exam for fourth-year students. 45% of 121 items had at least one "Maybe" rating. About 10% of all ratings were in the "Maybe" category. Case failure rates varied considerably depending on the simulated severity of ratings. Overall failure rates were not substantially impacted. The three-level Angoff retains the cognitive simplicity of the Yes/No Angoff while addressing the challenge of items midrange for the borderline candidate and avoids the potential bias.